Semantic representation module of a machine-learning engine in a video analysis system

ABSTRACT

A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/921,595, filed Mar. 14, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/494,010, filed Apr. 21, 2017, now U.S. Pat. No.9,946,934, issued Apr. 17, 2018, which is a continuation Ser. No.15/338,072, filed Oct. 28, 2016, now U.S. Pat. No. 9,665,774 issued May30, 2017; which is a continuation of U.S. patent application Ser. No.14/992,973, filed Jan. 11, 2016 now U.S. Pat. No. 9,489,569, issued Nov.8, 2016; which is a continuation of U.S. patent application Ser. No.14/584,967, filed Dec. 29, 2014, now U.S. Pat. No. 9,235,752, issuedJan. 12, 2016; which is a continuation of U.S. patent application Ser.No. 13/855,332, filed Apr. 2, 2013, now U.S. Pat. No. 8,923,609, issuedDec. 30, 2014; which is a continuation of U.S. patent application Ser.No. 12/170,268, filed Jul. 9, 2008, now U.S. Pat. No. 8,411,935 issuedApr. 2, 2013; which claims priority to U.S. provisional application60/949,107, filed Jul. 11, 2007, the contents of each of which areincorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention generally relates to video analysis, and moreparticularly to analyzing and learning behaviors based on streamingvideo data.

Description of the Related Art

Some currently available video surveillance systems have simplerecognition capabilities. However, many such surveillance systemsrequire advance knowledge (before a system has been developed) of theactions and/or objects the systems have to be able to seek out.Underlying application code directed to specific “abnormal” behaviorsmust be developed to make these surveillance systems operable andsufficiently functional. In other words, unless the system underlyingcode includes descriptions of certain behaviors, the system will beincapable of recognizing such behaviors. For example, monitoring airportentrances for lurking criminals and identifying swimmers who are notmoving in a pool are two distinct situations, and therefore may requiredeveloping two distinct software products having their respective“abnormal” behaviors pre-coded. Further, for distinct behaviors,separate software products often need to be developed. This makes thesurveillance systems with recognition capabilities labor intensive andprohibitively costly.

Surveillance systems may also be designed to memorize normal scenes andgenerate an alarm whenever what is considered normal changes. However,these types of surveillance systems must be pre-programmed to know howmuch change is abnormal. Further, such systems cannot accuratelycharacterize what has actually occurred. Thus, products developed insuch a manner are configured to detect only a limited range ofpredefined type of behavior.

SUMMARY OF THE INVENTION

Embodiments of the invention provide a machine-learning engineconfigured to recognize and learn behaviors, as well as to identify anddistinguish between normal and abnormal behavior within a scene, byanalyzing movements and/or activities (or absence of such) over time.

One embodiment of the invention includes a method for processing datadescribing a scene depicted in a sequence of video frames. The methodmay generally include receiving input data describing one or moreobjects detected in the scene. The input data includes at least aclassification for each of the one or more objects. This method may alsoinclude identifying one or more primitive events, where each primitiveevent provides a semantic value describing a behavior engaged in by atleast one of the objects depicted in the sequence of video frames andwherein each primitive event has an assigned primitive event symbol. Themethod may still further include generating, for one or more objects, aprimitive event symbol stream which includes the primitive event symbolscorresponding to the primitive events identified for a respective objectand generating, for one or more objects, a phase space symbol stream.The phase space symbol stream describes a trajectory for a respectiveobject through a phase space domain. This method may also includecombining the primitive event symbol stream and the phase space symbolstream for each respective object to form a first vector representationof that object and passing the first vector representations to a machinelearning engine configured to identify patterns of behavior for eachobject classification from the first vector representation.

Still another embodiment includes a method for processing data generatedfrom a sequence of video frames. This method may generally includereceiving, as a trajectory for a first object, a series of primitiveevents associated with a path of the first object depicted in thesequence of video frames as the first object moves through the scene.Each primitive event includes at least an object type and a set of oneor more kinematic variables associated with the second object. Afterreceiving the trajectory for the first object, a first vectorrepresentation generated for the first object is received. The firstvector representation may be generated from a primitive event symbolstream and a phase space symbol stream. Typically, these streamsdescribe actions of at least the first object depicted in the sequenceof video frames. This method may also include exciting one or more nodesof a perceptual associative memory using the trajectory and the firstvector representation and identifying, based on the one or more excitednodes, a percept. Once identified, the percept may be copied to aworkspace. In response to the particular percept (or precepts) copied tothe workspace, a codelet is selected and invoked. The codelet mayinclude an executable sequence of instructions.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features, advantages andobjects of the present invention are attained and can be understood indetail, a more particular description of the invention, brieflysummarized above, may be had by reference to the embodiments illustratedin the appended drawings.

It is to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is a block diagram illustrating a behavior recognition system,according to one embodiment of the invention.

FIG. 2 illustrates a machine learning engine, according to oneembodiment of the invention.

FIG. 3 illustrates a flowchart of a method for analyzing, learning, andrecognizing behaviors, according to one embodiment of the invention.

FIG. 4 illustrates a semantic representation module of a machinelearning engine, according to one embodiment of the invention.

FIG. 5 illustrates a trajectory of an object/subject through aphase-space domain, according to one embodiment of the invention.

FIG. 6 illustrates a flowchart of a method for providing semanticrepresentations of behaviors, according to one embodiment of theinvention.

FIG. 7 illustrates a perception module of a machine learning engine,according to one embodiment of the invention.

FIGS. 8A-8C illustrate a flowchart of a method analyzing, learning, andrecognizing behaviors, according to one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Machine-learning behavior-recognition systems learn behaviors based oninformation acquired over time. In context of the present invention,information from a video stream (i.e., a sequence of individual videoframes) is analyzed. Embodiments of the invention provide amachine-learning engine configured to recognize and learn behaviors, aswell as to identify and distinguish between normal and abnormal behaviorwithin a scene, by analyzing movements and/or activities (or absence ofsuch) over time. Normal/abnormal behaviors need not all be pre-definedor hard-coded. Instead, the machine-learning engine described hereinrapidly learns what is “normal” for a given environment and identifiesabnormal behaviors based on what is learned through monitoring thescene, i.e., by analyzing the content of recorded video, frame-by-frame.

In one embodiment, a computer vision engine is connected with a machinelearning engine. Generally, the computer vision engine receives an inputvideo stream and analyzes the stream frame-by-frame to identity objectsand scene topography, to distinguish background elements of the scenefrom foreground elements, etc. As the computer vision engine “sees”these types of things and events occurring in the scene, thisinformation may be input to the machine learning engine. In turn, themachine learning engine may include a semantic analysis model and acognitive model. The semantic analysis model may label events observedby the computer vision engine with semantic meaning. That is, thesemantic analysis model may identity what the tracked elements in thescene are doing. The cognitive model may be configured to identifypatterns of behavior, leading to a “learning” of what events occurwithin a scene. Thus, the cognitive model may, over time, developsemantic labels to apply to observed behavior. In one embodiment, thesystem provides for progressive levels of complexity in what may belearned from the scene. For example, combinations of primitive events“seen” by the computer vision engine may be labeled as instances of ahigher-order behavior, e.g., the primitive events of “car enters scene,”“car moves to location A,” and “car stops” might be labeled as “parking”by the semantic analysis model. In turn, sequences of such instances maythemselves be labeled as instances of yet another higher-order behavior,and so on. Further, as these events are observed (and labeled) themachine learning engine may identity which ones fall into a range ofexpected behaviors for a scene and which ones represent an unusual (ornew) pattern of behavior. The machine learning engine may be configuredto generate alerts (or perform some other predefined action) whencertain events are observed.

In the following, reference is made to embodiments of the invention.However, it should be understood that the invention is not limited toany specifically described embodiment. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theinvention. Furthermore, in various embodiments the invention providesnumerous advantages over the prior art. However, although embodiments ofthe invention may achieve advantages over other possible solutionsand/or over the prior art, whether or not a particular advantage isachieved by a given embodiment is not limiting of the invention. Thus,the following aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,reference to “the invention” shall not be construed as a generalizationof any inventive subject matter disclosed herein and shall not beconsidered to be an element or limitation of the appended claims exceptwhere explicitly recited in a claim(s).

One embodiment of the invention is implemented as a program product foruse with a computer system. The program(s) of the program productdefines functions of the embodiments (including the methods describedherein) and can be contained on a variety of computer-readable storagemedia. Illustrative computer-readable storage media include, but are notlimited to: (i) non-writable storage media (e.g., read-only memorydevices within a computer such as CD-ROM disks readable by a CD-ROMdrive) on which information is permanently stored; (ii) writable storagemedia (e.g., floppy disks within a diskette drive or hard-disk drive) onwhich alterable information is stored. Such computer-readable storagemedia, when carrying computer-readable instructions that direct thefunctions of the present invention, are embodiments of the presentinvention. Other media include communications media through whichinformation is conveyed to a computer, such as through a computer ortelephone network, including wireless communications networks. Thelatter embodiment specifically includes transmitting information to andfrom the Internet and other networks. Such communications media, whencarrying computer-readable instructions that direct the functions of thepresent invention, are embodiments of the present invention. Broadly,computer-readable storage media and communications media may be referredto herein as computer-readable media.

In general, the routines executed to implement the embodiments of theinvention may be part of an operating system or a specific application,component, program, module, object, or sequence of instructions. Thecomputer program of the present invention is comprised typically of amultitude of instructions that will be translated by the native computerinto a machine-readable format and hence executable instructions. Also,programs are comprised of variables and data structures that eitherreside locally to the program or are found in memory or on storagedevices. In addition, various programs described herein may beidentified based upon the application for which they are implemented ina specific embodiment of the invention. However, it should beappreciated that any particular program nomenclature that follows isused merely for convenience, and thus the invention should not belimited to use solely in any specific application identified and/orimplied by such nomenclature.

Embodiments of the present invention provide a machine learning enginefor a behavior recognition system and a method for analyzing, learning,and recognizing behaviors. FIG. 1 is a block diagram illustrating abehavior recognition system 100, according to one embodiment of thepresent invention. As shown, the behavior recognition system 100includes a video input 105, a network 110, a computer system 115, andinput and output devices 145 (e.g., a monitor, a keyboard, a mouse, aprinter, and the like).

The network 110 receives video data (e.g., video stream(s), videoimages, or the like) from the video input 105. The video input 105 maybe a video camera, a VCR, DVR, DVD, computer, or the like. For example,the video input 105 may be a stationary video camera aimed at certainarea (e.g., a subway station) that continuously records the area andevents taking place therein. Generally, the area visible to the camerais referred to as the “scene.” The video input 105 may be configured torecord the scene as a sequence of individual video frames at a specifiedframe-rate (e.g., 24 frames per second), where each frame includes afixed number of pixels (e.g., 320×240). Each pixel of each framespecifies a color value (e.g., an RGB value). Further, the video streammay be formatted using known such formats e.g., MPEG2, MJPEG, MPEG4,H.263, H.264, and the like. The network 110 may be used to transmit thevideo data recorded by the video input 105 to the computer system 115.The behavior recognition system 100 analyzes this raw information toidentify active elements in the stream, classifies such elements,derives a variety of metadata regarding the actions and interactions ofsuch elements, and supplies this information to a machine learningengine 140. As described in greater detail below, the machine learningengine 140 may be configured to evaluate the received information andremember the received information and results of the evaluation overtime. Further, the machine learning engine may identify certainanomalous and/or normal behaviors.

Illustratively, the computer system 115 includes a CPU 120, storage 125(e.g., a disk drive, optical disk drive, floppy disk drive, and thelike), and memory 130 containing a computer vision engine 135 and themachine learning engine 140. The computer vision engine 135 may providea software application configured to analyze a sequence of video framesprovided by video input 105. For example, in one embodiment, thecomputer vision engine 135 may be configured to analyze video frames toidentify targets of interest, track those targets of interest, inferproperties about the targets of interest, classify them by categories,and tag the observed data. In one embodiment, the computer vision engine135 generates a list of attributes (such as texture, color, and thelike) of the classified objects of interest and provides the list to themachine learning engine 140. Additionally, the computer vision engine135 may supply the machine learning engine 140 with a variety ofinformation about each tracked object within a scene (e.g., kinematicdata such as time, position, velocity, etc., data, color, data,appearance data, etc.).

In one embodiment, the machine learning engine 140 receives the videoframes and the results generated by the computer vision engine 135. Themachine learning engine 140 analyzes the received data, builds semanticrepresentations of behaviors/events depicted in the video frames andlearned over time, determines patterns, and learns from these observedbehaviors to identify normal and/or abnormal events. Data describing anormal (or abnormal) behavior/event, along with the semantic labelsapplied to such an event, may be provided to an output devices 145 toissue alerts, e.g., an alert message presented on a GUI interfacescreen.

The computer vision engine 135 and the machine learning engine 140 mayeach be configured to process the received video data, generally, inreal-time. That is, the computer vision engine 135 may be configured to“see” events as they occur, e.g., to identify and track an object movingabout the scene. However, the machine learning engine 140 (i.e., asemantic model and a cognitive model) may lag behind in evaluating thesequence of event being observed by the computer vision engine. Thus,time scales for processing information by the computer vision engine 135and the machine learning engine 140 may differ. For example, in oneembodiment, the computer vision engine 135 processes the received videodata frame by frame, while the machine learning engine processes thereceived data every N-frames.

Note, however, FIG. 1 illustrates merely one possible arrangement of thebehavior recognition system 100. For example, while the video input 105is shown connected to the computer system 115 via the network 110, thenetwork 110 is not always present or needed (e.g., the video input 105may be directly connected to the computer system 115). Further, in oneembodiment, the computer vision engine 135 may be implemented as a partof a video input device (e.g., as a firmware component wired directlyinto a video camera). In such a case, the outputs of the video cameramay be provided to the machine learning engine 140 for analysis.Moreover, while the machine learning engine 140 is depicted as a part ofthe computer system 115, it may be implemented as a system separate fromthe computer system 115 that communicates with the computer system 115via the network 110 or, alternatively, as a part of a different system.

FIG. 2 illustrates a machine learning engine 140, according to oneembodiment of the invention. Generally, the machine learning engine 140employs two models for recognizing, analyzing, and learning behaviors;namely, a semantic model and a cognitive model. Based on data providedby the computer vision engine, the semantic model generates semanticdescriptions (representations) of what is depicted in the video streamincluding semantic descriptions (representations) of objects/subjectsand their actions. In other words, the semantic model provides labelsdata with semantic meaning as to what is observed in the scene. In turn,the cognitive model may be configured to observe patters associated witha given event; update a pattern (i.e., a memory) representing a givenevent; reinforcing long-term memories associated with an event; develop“memories” representing new patterns of behavior; create new semanticlabeling to apply to new patterns of behavior. As stated, in oneembodiment, new patterns of behavior may be generated as a combinationof known patterns. In such a case, the semantic labeling applied to anew behavior may represent a combination of the labels applied topatterns in that new behavior.

Thus, the cognitive model may simulate some aspects of a human brain,e.g., how the human brain perceives abstract concepts, reasons aboutthem, recognizes behaviors, and learns new concepts. In one embodiment,the cognitive model may employ a neuro-semantic network that includes acombination of a semantic representation module 205 and a cognitivemodel 225. Each of these components is described in greater detailbelow. The neuro-semantic network may include a plurality of nodesrepresenting semantic concepts (i.e., a neural net). As is known, aneural net may represent simple concepts using a single node (e.g., avehicle or its kinematic characteristic) and complex concepts may berepresented by multiple nodes that includes multiple concepts connectedby links (e.g., motor-vehicle accident). The neuro-semantic network mayinclude several levels, where the lowest level describes a collection ofprimitive events. Higher levels of the neuro-semantic network maydescribe complex concepts, which are created by combining primitiveconcepts. Typically, the higher the level of complexity, the morecomplex concepts it defines. In one embodiment, the neuro-semanticnetwork may provide increasing levels of complexity where the primitivesfor one level of complexity are combined to form a primitive for thenext level of complexity, and so on. Data provided to the cognitivemodel may be used to excite nodes of the neuro-semantic network,allowing behaviors to be recognized and the network itself to beupdated. Updates may include creating nodes, updating nodes, deletingnodes or modifying or creating links between nodes.

In one embodiment, the semantic representation module 205 receives datadescribing objects/subjects detected in a scene from the computer visionengine 135. Such data may include identification data, posture,location, trajectory, velocity, acceleration, direction, and otherquantitative characteristics that describe an object identified in thescene by the computer vision engine 135. Based on data received from thecomputer version engine 135, the semantic representation module 205forms two semantic streams; namely, a primitive event symbol stream anda phase-space symbol stream. The primitive event symbol stream includessemantic i.e., symbolic, descriptions of primitive events recognized inthe scene and objects participating in such primitive events (e.g.,“vehicle stops,” “human turns,” etc.). The phase-space partitioningstream includes semantic descriptions, i.e., phase-space symbols, ofvalues of quantitative characteristics of an object (e.g., a symbol “a”indicating that an object was located in a certain area of the scene ora symbol “x” indicating that an object's velocity is within a certainrange, and so on). Thus, the phase-space symbol stream associated with agiven object may provide a trajectory of motion for that objectthroughout the scene. The semantic representation module 205 maygenerate formal language vectors based on the trajectories of a givenobject by combining relative data from the primitive event andphase-space symbol streams. As described in greater detail herein, theformal language vectors are used to describe both semantic andquantitative aspects of behavior observed to have occurred within ascene.

As shown, the cognitive model 225 includes a perception module 210, abehavior comprehension module 215, and reinforcement and decay module220. In general, the perception module 210 analyzes data provided by thesemantic representation module 205, learns patterns, generalizes basedon observations, and learns by making analogies. In one embodiment, theperception module 210 may include multiple memories such as a perceptualmemory, an episodic memory, and a long-term memory. Based on theincoming data, the perception module 210 may perceive multi-levelconcepts (structures), such as a percept. As used herein a “percept”represents a combination of nodes (and links between nodes) representingan action and/or associated actor(s); trajectories, i.e., sequences ofpercepts; and clusters of trajectories. That is, a percept may bedefined as a subgraph of a neural net that includes each node (and linksbetween node) relevant for a particular identified behavior. Thus,percepts may represent behaviors perceived by the machine learningengine to have occurred. More complex behaviors may be represented ascombinations of percepts. As described in greater detail below,perceived concepts and corresponding memories may be stored in aworkspace and processed by various codelets. In one embodiment, acodelet provides an active, typically independent, process (agent) thatincludes executable code. Generally, a codelet may evaluate percepts andrelationships between percepts to recognize behaviors and other eventsimportant to the system (e.g., a parking event), build new structuresbased using analogies (e.g., combine two similar percepts into a higherlevel node), detect anomalies (e.g., by comparing percepts to long-termmemory content), look for expected events/behaviors, and so on.

In one embodiment, the perception module 210 may be further configuredto determine whether the computer vision engine 135 has misclassified anobject. For example, if the perception module 210 determines that thecomputer vision engine has repeatedly applied particular classificationto an object (e.g., a car) and then classifies this same object assomething else (e.g., a person), the perception module 210 may informthe computer vision 135 of the misclassification.

In general, the behavior comprehension module 215 recognizes behaviorsand responds to recognized behaviors. For this purpose, the behaviorcomprehension module 215 further analyzes structures placed in theworkspace. As the presence of given percepts are broadcast to othercomponents of the cognitive model 225, multiple internal and externalactions may be performed. For example, internal actions may includeupdating and/or generalizing procedures and concepts, models and events,creating new concepts and procedures, generating expectationstructures/procedures, and so on. In one embodiment, external actionsmay include issuing a signal (e.g., alarm) responsive to recognized (orunrecognized) behavior, providing feedback to other components of thebehavior recognition system 100 (such as the semantic representationmodule 205, the computer-vision engine 135, etc.), adjusting cameraoperations, and so on. The feedback may include data regarding theobserved events/behaviors needed to modify the behavior-recognitionsystem to better recognize the events/behaviors in the future.

In general, the reinforcement and decay module 220 reinforces memoriesof repeatedly occurring behaviors and decays and/or eliminates memoriesof occasionally occurring behaviors. More specifically, percepts, andassociated nodes, may decay over time if not used or alternatively, maybe reinforced, if used. Thus, for example, when a structure, such as apercept, is placed into the workspace similar memories may be reinforced(or updated to better generalize the behavior represented by thememory). In this manner, a competitive learning environment is createdwhere useful percepts, and associated nodes, survive because they arereinforced, and non-useful, percepts, and associated nodes, decay away.

FIG. 3 illustrates a flowchart of a method 300 for analyzing, learning,and recognizing behaviors observed in an input video stream, accordingto one embodiment of the invention. The method 300 starts at step 305.At step 310, the machine learning engine 140 receives data describingthe objects detected in the scene by the computer vision engine. Asdiscussed above, such data may include an objects' dynamic and kinematiccharacteristics (e.g., time, position, velocity, etc.), identificationinformation, classification information, and so on. And further, thedata may be received in a generally real-time stream as the computervision engine processes a video stream, frame-by-frame.

In one embodiment, the received data is used to generate create two datastreams, a primitive event symbol stream and a phase-space symbol stream(step 315). The primitive event symbol stream includes semanticrepresentations of the detected primitive events. For example, a streamof primitive events related to behavior the computer vision engine“sees” as a vehicle parking could include “vehicle appears,” “vehiclemoves,” “vehicle turns,” and “vehicle stops.” The phase-space symbolstream includes symbolic representations of values of objects/subjects'quantitative characteristics, such as location, time, velocity, and soon. For example a phase-space symbol stream corresponding to theprimitive event stream of the example above may be the following: “a, c,f, k,” where each of the symbols corresponds to a region of the scenewhere a particular primitive event took place, i.e., the vehicleappeared in area “a,” moved through area “c,” turned in area “f,” andstopped in area “k.” Though, the provided example includes only oneobject, i.e., the vehicle, each stream typically includes similarinformation describing other objects/subjects detected in the scene.Alternatively, multiple pairs of primitive event and phase-space symbolstreams may be generated, namely a pair of streams for each detectedobject/subject.

As an object moves around the scene, the computer vision enginegenerates a trajectory along a path of movement of that object. In oneembodiment, such a trajectory may be used to organize semanticrepresentations, which relate to one object into one entity, referred toas a “formal language vector.” At step 320, data from the primitiveevent and phase-space symbol streams related to each object having acomplete trajectory is combined and converted to generate respectiveformal language vectors. The vector representations, along with theprimitive event streams may be passed to a perceptual memory. Forexample, the formal language vectors may be passed to the cognitivemodel 225 of the machine-learning engine 140 for analysis.

Typically, a trajectory starts when an object or subject appears in thescene, continues while the object moves about the scene, and iscompleted when that object disappears from the scene. In one embodiment,the trajectory may be considered to be complete when an object/subjectstops moving for a long period of time (e.g., if a car arrived into aparking lot, its trajectory would be completed when the car parks andstays parked for a period of time). A period of time may be defined, forexample, by a number of video frames. Further, in one embodiment, theappropriate amount of time may be “learned” by the system as a matter ofobservation. Accordingly, a trajectory may also begin when anobject/subject that has been motionless for a period of time startsmoving. In another embodiment, a trajectory is not completed if anobject/subject disappears from the scene only temporarily (e.g., anobject passes behind a background object such as a tree). If after sucha temporary disappearance, the object continues to be identified as thesame object, its trajectory continues until the object fully disappearsfrom the scene, or alternatively, stops for a period of time. Note,although a trajectory, as described above, is defined in terms of anobject (or subject) appearing in or disappearing from the scene, atrajectory may also be defined in different terms. Such terms coulddepend on, for example, the particular characteristics of the observedenvironment and/or the requirements of a particular case. For example,in one embodiment, a trajectory may be defined in terms of time (e.g.,trajectory occupies a pre-defined number of video frames).

At step 325, the incoming primitive event and phase-space symbol streamsand/or formal language vectors may be used to excite nodes in aneuro-semantic network of a perceptual memory of the cognitive model225. In one embodiment, the neuro-semantic network is a directed graphthat includes nodes (i.e., vertices) representing concepts (e.g., nodeshave assigned concept labels) and links (i.e., edges) representing arelationship between two concepts (e.g., links have assignedrelationship labels). The nodes of the neuro-semantic network may beactivated by an appropriate stimulus (e.g., input from the computervision engine regarding what objects are “seen” in the scene). Also, thestimulation of one mode may be iteratively propagated to other nodes.That is, exciting one node may excite another. And the stronger a linkbetween two nodes, the more an excitation of one node may excite theother. In one embodiment, activation of one node is propagated to thelinked nodes only when the activation value of that node reaches apre-defined activation threshold. Further, activation values may decayas the activations are propagated through the neuro-semantic network. Inother words, a node from which the activation is propagated would have ahigher activation value than nodes to which the activation is propagatedto, such as conceptually linked nodes. Thus, at some point afterreceiving input from the computer vision engine, a given, a set ofexcited nodes responsive to the input may be identified. The set ofexacted nodes generally represents a percept, i.e., what is perceived bythe perceptual memory in response to the input.

As described above, the neuro-semantic network may include a pluralityof levels, where the lowest level represents the simplest concepts, suchas semantics of primitive events (e.g., “vehicle,” “turns,” etc.). Thecollections of nodes (and links between nodes) represent more complexconcepts, for example, concepts involving multiple primitive events(e.g., parking), where such primitive events are represented by lowerlevel nodes. In this manner, higher level concepts are linked to thelower level primitive events. As data, representing primitive events isreceived, the corresponding nodes are activated, and their activation ispropagated to the conceptually linked nodes. Note however, thatdifferent higher level nodes may be conceptually linked to the samelower level nodes, e.g., nodes representing primitive events “vehiclestops” and “vehicle moves” may both be conceptually linked to the samenode “vehicle.”

At step 330, a collection of nodes excited by the data received at step325 may be copied into a workspace. The workspace may provide a datastructure used by the process and analyze data as events are observed bythe memories of the machine learning engine. Items in the workspace atany given time may be said to receive the “focus of attention” of themachine learning engine. Typically, nodes or combinations of nodes(i.e., percepts) copied into the workspace represent behaviors currentlyobserved in the scene. In one embodiment, data stored in a long termmemory, episodic memory, and/or perceptual memory of the cognitive model225 may also be copied to the workspace. That is, once a perceptrepresenting a current event is passed into the workspace, the machinelearning engine 140 may identify similar memories used by a codelet tocompare the current event with past experience. In one embodiment, thedata may be represented using a structure similar to the percept. Thatis, the memories may be represented in the workspace as a directed graphof nodes. Once the relevant data is in the workspace, codelets mayprocess and analyze data representing the currently observed behaviorsand relate it to past behaviors (represented by memories copies from theepisodic and/or long term memories).

Thus, the workspace generally allows codelets to analyze what isobserved in the scene. To achieve this, codelets evaluate percepts andrelationships between the percepts. More specifically, codelets mayidentify input features and create semantic events; connect percepts(nodes) in the workspace; determine expected events based on a sequenceof percepts; determine anomalies; look for expected outcomes andindicate when the expected outcomes do not occur (and adjustexpectations accordingly); and so on. The codelets may also increase abond between two (or more) percepts, build new percepts (e.g., byanalogy), destroy existing percepts, execute other codelets, etc.

At any particular time, numerous codelets may be in the workspaceawaiting execution. Accordingly, at step 335, a codelet is selectedamong the available codelets. In one embodiment, codelets may beassigned a weighted value (e.g., events requiring immediate attentionmay be given greater weight for execution then others). In such a case,a codelet may be selected in a semi-random manner, where a codelethaving a higher weighting is more likely to be selected than a codelethaving a lower weighting (or no weighting).

As codelets are executed, some percepts may become excited. For example,a “parking” codelet could be selected to evaluate formal languagevectors labeled as representing a parking event. In doing so, the“parking” codelet could retrieve memories (both long term and episodic)representing other occurrences of the “parking” behavior and compare theretrieved memories to the new instance of the parking event representedby the formal language vectors.

The percepts copied into the workspace may be reinforced at step 340.Alternatively, percepts stored in the memory structures (e.g., theperceptual memory and the episodic memory) that are not acted upon maydecay over time, and eventually be eliminated. This should make sense asthe perception module 210 includes a memory used to perceive events asthey occur. Thus, memories in perception module 210 should decay overtime. That is, as new events are “perceived” older events may decayaway. At the same time, features related to a given event may be storedin episodic memory (essentially a short term memory of specific, recentevents) and also used to reinforce or adjust long-term memories. Forexample, long term memories of “parking” would not include any referenceto a particular car identified and tracked in a scene, where theepisodic memory would retain this information for a period of time. Atthe same time, the long term memory could “remember” that cars park in aparticular location within the scene. Thus, if a car was perceived as“parking” in at a location different from any location where cars havepreviously parked—the event could be identified as an anomaly. However,if the same “anomaly” continued to occur, a long term memory of a newparking location would develop. In other words, at step 340 memoriescorresponding to repeatedly occurring behaviors are reinforced (andupdated to capture different variations of the semantic event, e.g.,different instances of parking), while memories corresponding tooccasionally occurring behaviors decay or are eliminated. Note however,that different types of percepts/codelets may decay at different speeds.For example, in one embodiment, percepts and codelets associated withabnormal behaviors decay slower than percepts and codelets associatedwith normal behaviors. Furthermore, in another embodiment, decay is notlinear. For example, if a certain precept/codelet has been reinforcedabove a certain threshold, then such a percept or codelet would decayslower than if it did not reach the threshold. After the memories havebeen reinforced and/or decayed, the method returns to step 310.

Note however, that it is not necessary to perform all of theabove-described steps in the order named. Furthermore, not all of thedescribed steps are necessary for the described method to operate. Whichsteps should be used, in what order the steps should be performed, andwhether some steps should be repeated more often than other steps isdetermined, based on, for example, needs of a particular user, specificqualities of an observed environment, and so on. For example, though atstep 335, as described, only one codelet is selected before the memoriesare reinforced or decayed, in another embodiment, multiple codelets areselected and executed before the memories are decayed and/or reinforced.

FIG. 4 illustrates an example of a semantic representation module 205 ofa machine learning engine 140, according to one embodiment of theinvention. As shown, the semantic representation module 205 includes asensory memory 405, a latent semantic analysis (LSA) training module410, a primitive event detection module 415, a phase space partitioningmodule 420, an incremental latent semantic analysis (i-LSA) updatemodule 430, and a formal language learning module 440. Generally, thesemantic representation module 205 creates semantic representations ofmotions and actions of the objects/subjects observed and tracked in thescene. The semantic representations provide a formal way to describewhat is believed to be happening in the scene based on motions of aparticular tracked object/subject (and ultimately, based on changes inpixel values from frame-to-frame). Subsequently, the semanticrepresentations are provided to the perception module 210 and analyzedfor recognizable patterns, i.e., the perception model 210 is generallyconfigured to perceive what is occurring in the scene.

In on embodiment, the sensory memory 405 acquires data provided to thesemantic representation module 205 by the computer vision engine 135 andstores this data for subsequent use by the primitive event detectionmodule 415 and the phase-space partitioning module 420. That is, thesensory memory 405 may provide a buffer for trajectory points ofobjects/subjects and object/subjects' characteristics, such as time,velocity, acceleration, and so on. The sensory memory 405 may be viewedas a catalog of actions/events that have been recently observed by thebehavior-recognition system 100. Such actions/events are stored in thesensory memory 405 for brief periods of time, e.g., in one embodiment, afew seconds. The sensory memory 405 may also select what information toprovide to the primitive event detection module 415 and the phase spacepartitioning module 420 and/or receive feedback from the primitive eventdetection module 415.

In general, the primitive event detection module 415 is configured toidentify the occurrence of primitive events (e.g., vehicle stops,reverses direction, disappears, appears; person bends, falls; exchange,and the like) using information provided by the sensory memory 405.Typically, the primitive events reflect changes in kinematic/dynamiccharacteristics of the tracked objects/subjects. Accordingly, theprimitive event detection module 415 analyzes the kinematic (dynamic)and/or posture data associated with the tracked objects/subjects andprocesses such data into components having assigned values (e.g., one ormore symbols) representing primitive events and activities. In oneembodiment, such data also includes numerical data obtained from sensorsof a video acquisition device. Of course, the range of primitive eventsis not limited to the behavior of a vehicle; rather a set of primitiveevents may be defined for each object that may be identified and trackedby the computer vision engine. For example, assume a computer visionengine configured to classify tracked objects as being a “vehicle,” a“person,” an “other,” or an “unknown.” In this example, “unknown” couldrepresent an object classified as either a “vehicle” or a “person,” butclassified without sufficient confidence to select between the two,where “other” could be used to classify an object as affirmatively notbeing a “vehicle” or a “person.” In such a case, different sets ofprimitive events may be available to describe the behavior of eachdifferent type of object. Of course, the objects recognized orclassified by the computer vision engine may be configured to suit theneeds of a particular case, and the classifications of “vehicle,”‘person,” “unknown,” and “other” are provided as an illustrativeexample.

Typically, a formal language grammar (e.g., nouns and verbs) is used todescribe the primitive events (e.g., “car parks,” “person appears,” andthe like) the nouns relate to objects identified by the computer visionengine and the verbs come from actions performed by the object andperceived by the semantic representation module 205. Thus, for example,once an object is classified as being a “vehicle,” the primitive eventdetection module 415 may evaluate data acquired about the object, i.e.,a car, to identify different behavioral events as they occur and emitappropriate symbols into a primitive event symbol stream (e.g., “vehicleappears,” “vehicle moves,” “vehicle turns,” “vehicle stops,” “vehicleparks,” etc.). Further, various activities, such as postures andgestures, may be implemented in primitive event detection module 415 toprovide feature information in the primitive event symbol stream.Further the machine learning engine may, over time, develop memoriesrepresenting combinations of objects and symbol streams, and new eventsmay be compared with the memories, used to reinforce or update memories,etc.

In one embodiment, to identify a primitive event associated with anobject/subject, a state machine may be assigned to the object/subject sothat only event detection algorithms satisfying the kinematics of thecurrent object/subject's state need to be run at each particular step.Such states may, for example, include stationary state (object/subjectstopped), moving state (object/subject moves), and unknown state(observed action does not satisfy definitions of any of the otherstates). Generally, the object/subject remains in a given state for afinite period of time and events corresponding to state transitions areinstantaneous (e.g., “starting” event, “stopping” event, etc.). However,some events may have duration over shorter time scale than a particularstate (e.g., “turning” event—“moving” state). Thus, in one embodiment,“turning” event is treated as instantaneous event corresponding totransition from a “moving” state to the “moving” state. Additionalinformation, such as turning angle, may be retained by the statemachine.

Note however, as implemented, a number of factors should be considered.For example, when analyzing a video frame having jitter, the stationarystate should not necessarily be determined based on only stationarypixels. Rather, allowances for the video jitter should be made.Accordingly, to determine object/subject's state numerous factors shouldbe considered, analyzed and estimated guesses should be made.

In one embodiment, the primitive event detection module 415 isconfigured to analyze only events involving single objects (e.g., “carmoves”). In another embodiment, the primitive event detection module 415may also analyze events involving multiple interacting objects (e.g.,interactions between two or more people). To decrease the amount of datain such embodiment, an interactive event may be considered only whenobjects/subjects possibly involved in the event are in proximity to eachother. The proximity measure may be defined, for example, by a number ofpixels.

The phase-space partitioning module 420 is generally configured todetermine symbolic representations of values of objects/subjects'quantitative characteristics, such as location, time, velocity, and soon, and emit the determined symbolic representations into a phase-spacesymbol stream. In one embodiment, the phase space partitioning module420 includes a physical description of the geometry of the scene. Thescene may be divided into a set of areas, where each area is assigned aphase-space symbol. The symbolic representations for a particular mobileobject (e.g., human, vehicle, etc.) may be formed in time order, as thetrajectory of the mobile agent is analyzed. When the mobile agent entersa given area, the corresponding phase-space symbol is emitted into thephase-space symbol stream.

The primitive event and phase-space symbol streams, created respectivelyby the primitive event detection module 415 and the phase-spacepartitioning module 420, are provided to the LSA training module 410 fortraining, or if the training has been completed, to the i-LSA updatemodule 430. In general, both the LSA training module 410 and i-LSAupdate module 430 analyze the incoming symbol streams and constructvector representations (e.g., formal language vectors) of theevents/behaviors observed in the scene. In one embodiment, the LSAtraining module 410 and the i-LSA update module 430 also use, construct,and/or update clusters of behavior vectors, where each cluster typicallyrepresents a pattern corresponding to a known behavior.

The LSA training module 410 may be configured to train the semanticrepresentation module 205 using data obtained from the computer visionengine. In one embodiment, the LSA training module 410 gathers dataregarding a scene until a layout for the scene is determined withsufficient statistical certainty. In other words, the LSA trainingmodule 410 learns basic layout of the scene (such as types/kinds ofbehaviors observed in the scene, the perceived geometry or dimension ofthe scene (e.g., size and depth of field measurements,), while i-LSAupdate module 430 incrementally updates such a layout, allowing theperceived layout of the scene to both improve over time as well asrespond to changes that may occur.

In one embodiment, data from the primitive event and phase-space symbolstreams is combined to form vector representations. Typically, eachvector representation includes data corresponding to a completetrajectory of an object tracked in the scene and represents a behaviorexhibited by that object/subject. A formed string of symbols (or set ofsymbol strings) corresponding to a given behavior is defined as thegrammar for that behavior.

In one embodiment, the LSA training module 410 and the i-LSA updatemodule 430 may generate low-dimensional vectors (i.e., formal languagevectors) using singular value decompositions (SVD) from the higherdimensional vectors generated by the semantic analysis. Similar specificbehaviors (e.g., parking a vehicle A and parking of a vehicle B in areaC) represented by low-dimensional vectors form a cluster of behaviorvectors corresponding to a certain type/pattern of behavior (e.g.,parking of a vehicle in area C). In one embodiment, such similarities(i.e., distances between the low-dimensional vectors) are used to definea similarity measure. The similarity measure may be used to compareincoming behaviors against the learned behaviors represented by theclusters of low-dimensional vectors. In this manner, the semanticrepresentation module 205 reduces kinematic and posture data receivedfrom a computer-vision engine 135 regarding objects tracked in the sceneinto a manageable size and format such that the data may be processed byother modules of the machine-learning engine 140.

The formal language learning module 440 may be generally configured tosupport and update a formal language model. The formal language modeldefines a formal language and grammars for a particular scene. Asdescribed above, the semantic representation module 205 providessemantic representations for detected primitive events and behaviors. Aparticular string of symbols and/or set of symbol strings may representa grammar of a particular primitive event or behavior. From the formallanguage stream formed by the i-LSA update module 430, the formallanguage model collects semantic representations of primitive events andbehaviors observed in a particular scene. In other words, the formallanguage model for a particular scene represents types/kinds of symbolicrepresentations and their combinations that may be generated for thatscene. As new primitive events and/or behaviors are recognized in thescene, the formal language learning module 440 updates the formallanguage model. Optionally, in one embodiment, the formal language modelmay be updated manually. In another embodiment, some structures of theformal language and/or grammars in the formal language model arepre-defined. The primitive event symbol stream and the formal languagestream may be provided to the perception module 210 for furtheranalysis.

FIG. 5 illustrates a trajectory of an object through a phase-spacedomain, according to one embodiment of the invention. As describedabove, the computer vision engine 135 may be configured to provide themachine-learning engine 140 with data regarding observed objects in thescene, e.g., quantitative characteristics, such as speed, acceleration,location, direction, time, and the like. The values for each suchcharacteristics create a corresponding domain of values for thatquantitative characteristic (e.g., location domain may include eachpixel of the scene, or alternatively, selected parts of the scene; speeddomain may include possible speed value for a specific kind of theobject, such as vehicle, human, etc., or alternatively, any other kindof the object/subject; and so on).

In one embodiment, quantitative characteristic domains (e.g., a locationdomain) are partitioned and assigned a unique symbol. For example, FIG.5 illustrates a phase-space domain, i.e., the location domain 500, whichincludes every pixel depicting the scene. Multiple partitions, such aspartitions 502, 504, 506, 508, 510, 512, and 514, are created andassigned unique symbols. Illustratively, the partition 502 is assignedsymbol “a,” the partition 504 is assigned symbol “b,” and so on. In oneembodiment, the domain partitions are simply created by dividing thedomain area into approximately equal parts (e.g., location domain'partitions containing the same number of pixels and similarly shaped).In another embodiment, partitioning of the domain may be based onspecific characteristics of a scene (e.g., location domain havingseparate partitions for each parking space in a parking lot).

As an object moves around the scene, quantitative characteristic valueschange. If such domains are partitioned and the partitions are assignedunique symbols (thus, forming phase-space domains), the movement of theobject through each of the domains may be characterized by a phase-spacesymbol string. For example, FIG. 5 shows a trajectory 520 correspondingto an object moving through the scene with the following phase-spacesymbol string: [a, b, k, k, i, i, c, c, d], where each symbol isdetermined frame-by-frame, based on the object/subject's location in thescene. Note however, though FIG. 5 illustrates partition of the locationdomain, domains of other quantitative characteristics may be partitionedin the similar manner.

FIG. 6 is a flowchart illustrating a method 600 for providing semanticrepresentations of behaviors, according to one embodiment of theinvention. The method starts at step 605. Step 610 provides for initialtraining of a module for forming the semantic representations ofbehaviors observed in the scene, such as the semantic representationmodule 205. More specifically, over a period of time, data describingbehaviors observed in the scene is collected, clusters of vectorsrepresenting similar observed behaviors are built, and a formal languagemodel is trained.

As described above, in one embodiment, each cluster of vectors mayrepresent a type/pattern of behavior that have been observed in thescene, where each vector is a low-dimensional vector representingvariations of the behavior type/pattern specific to a particularobject/subject tracked in the scene. In one embodiment, before anylow-dimensional vector is created, data describing at least severaltrajectories is generated and collected, i.e., vector representations.The collected vector representations are used to build a matrix which isdecomposed using SVD (singular value decomposition) to createlow-dimensional vectors. As the matrix is decomposed, a projection modelis built for projecting high dimensional vectors into low-dimensionalvectors. In other words, the SVD algorithm is applied to the vectorrepresentations, to reduce the size of the vectors to a smaller numberof dimensions. However, at the same time the SVD algorithm preserves asmuch relevant information as possible about relative distances betweenparticular behaviors (i.e., vector representations). Information lostduring such decomposition is, therefore, mostly noise. Consequently,similar behaviors become more similar, while dissimilar behavior becomemore distinct.

The formal language model includes language that could be used todescribe primitive events identified in the scene and grammars ofbehaviors that could be observed in the scene. In one embodiment, theformal language model is empty at the beginning of step 610.Alternatively, the formal language model may include some pre-definedlanguage and grammars.

Steps 615 through 660 represent functional steps performed after theinitial training has been completed. At step 615, objects identified inthe scene and data describing such objects is received (such as anobject ID, classification, posture, velocity, acceleration, trajectory,and other quantitative characteristics). At step 620, the data receivedat step 615 is evaluated to identify primitive events observed in thescene. In one embodiment, a primitive event symbol is used to representeach distinct primitive event, (e.g., a single character “a” or a stringof symbols describing an identified primitive event such as “humanbends”). Each identified primitive event is typically associated withone or more objects participating in that event. Thus, for example, aprimitive event of “human stops” could be associated with a particularindividual observed to have stopped moving within the scene. In oneembodiment, when a primitive event involves multiple objects (e.g.,interaction between two people) multiple primitive event symbols may begenerated, where each primitive event symbol is associated with one ofthe objects.

At step 625, a stream of primitive event symbols are updated for eachobject involved in the identified primitive events. As an object movesabout the scene, it may participate in a number of primitive events.Accordingly, in one embodiment, a sequence of primitive event symbolscorresponding to the primitive events that the object has participatedis maintained for each object in the scene. As new primitive events fora particular object are identified, the corresponding primitive eventsymbol is added to the primitive event stream for that object. In thismanner, when a trajectory of the object is complete, the primitive eventsymbol sequence semantically describes all the primitive events that theobject has participated in along its trajectory.

At step 630, a phase-space symbol stream may be generated from the datareceived at step 615. As described above, phase-space symbols mayprovide semantic representations of the characteristics identified for agiven object. In one embodiment, a phase-space symbol may be used toindicate locations within the scene. As the object moves about the sceneand participates (or engages) in behavior labeled as a primitive event,the object moves from one region of the scene to another. Accordingly,in one embodiment, a sequence of phase-space symbols recording anobjects trajectory through the scene may be maintained for each objectin the scene. As new data regarding the object is received, aphase-space symbol indicating a current location of an object may beadded to the phase-space symbol sequence of that object. That is, thephase-space symbol sequence is updated (step 635). In this manner, whena trajectory of the object is complete, the phase-space symbol sequencedescribes the trajectory of that object. In one embodiment, aphase-space symbol for an object is generated every time a primitiveevent symbol for that object has been generated. Doing so allows thesystem to track both what events occur, but where (or in whatphase-space) each event occurs.

At step 640, the semantic representation module may determine whetherany trajectories have been completed. If not, then the method 600proceeds to step 665. However, if a complete trajectory is identified,the method 600 proceeds to step 645. As described above, a completetrajectory typically represents the movement of an object in the scenefrom the moment the object appears to the time it leaves the scene.

At step 645, a semantic representation vector may be generated for eachcompleted trajectory. Generally, a semantic representation vector isgenerated as a combination of the primitive event and phase-space symbolstreams associated with a given object. For example, assume an objectclassified as a “vehicle” by the computer vision engine. When thevehicle completes its trajectory, i.e., leaves the scene, thecorresponding primitive event symbol stream could include followingprimitive event symbols: “appears,” “moves,” “turns,” “stops,” “parks,”“moves,” “disappears” and the phase-space symbol stream could includethe following phase-space symbols: “p1,” “p2,” “p3,” “p4,” “p5,” “p6⋅”In such a case, the semantic representation vector corresponding to thetrajectory could include all the above named symbols, namely [“p1,”“p2,” “p3” “p4,” “p5,” “p6⋅” “appears” “moves,” “turns,” “stops”“parks,” “moves,” “disappears;”].

At step 650, the formal language model and the clusters of vectorsrepresenting learned behaviors may be updated based on the semanticrepresentation vector(s) generated at step 645. For example, newlyidentified primitive event symbols and/or behavior grammars may be addedto the formal language model. At step 655, the low-dimensional vectorsare generated from the semantic representation vectors generated at step645, (i.e., formal language vectors) using the updated projection model.At step 660, the low-dimensional vectors along with the primitive eventand phase-space symbols may be passed to the other modules of thebehavior recognition system 100, e.g., the perceptual module 210. Atstep 655, the primitive events, including kinematic data such asposition, velocity, etc., are passed to other modules of the behaviorrecognition system 100. In one embodiment, the perceptual module 210uses the primitive events and associated kinematic data to excite nodesof a neuro-semantic network, notwithstanding whether new completetrajectories are available. When step 660 or 665 is completed, themethod 600 returns to step 615.

Note however, that though the above described method 600 referred onlyobjects (such as a car); the method 600 may be similarly applied tosubjects (such as a human) or combinations thereof. Further, the stepsdescribed above steps are not necessarily performed in the order named.Moreover, not all of the described steps are necessary for the describedmethod to operate. Which steps should be executed, in what order thesteps should be executed, and whether some steps should be repeated moreoften than other steps is determined, based on, for example, needs of aparticular user, specific qualities of an observed environment, and soon.

FIG. 7 illustrates components of a perception module 210 of a machinelearning engine 140, according to one embodiment of the invention. Theperception module 210 may be generally configured to identify recurringpatterns of behavior, generalize such patters based on observations(i.e., to create memories), and learn by making analogies. In otherwords, the perception module 210 assigns meaning to, and learns from,incoming sensory data supplied by the semantic representation module205. As shown in FIG. 7, the perception module 210 may include aperceptual associative memory 705, an episodic memory 710, a long-termmemory 715, a scheduler 720, and a workspace 725. The workspace 720provides a data structure that represents information currently beingevaluated by the machine learning engine 140. That is, at any givenmoment, the workspace 725 stores elements of data that currently havethe “focus-of-attention” of the machine learning engine 140. Theworkspace 725 may include precepts and codelets relevant to what hasbeen observed to be occurring in the scene at any given point in time.As described above, codelets may be activated and applied to percepts inthe workspace. In one embodiment, the codelets indicate which data fromthe workplace 725 should be provided to the behavior comprehensionmodule 215 for further analysis.

In one embodiment, the perceptual memory 705 may be configured to assigninterpretations made by the perception module 210 to incoming stimuli.In general, the perceptual associative memory 705 collects data providedto the perception module 210 and stores such data as percepts. Morespecifically, in one embodiment, the perceptual associative memory 705is implemented as a neuro-semantic network containing a plurality ofnodes, each representing a semantic concept and links representingrelationships between the concepts.

As described above, nodes of the neuro-semantic network may be excitedby an appropriate stimulus (e.g., data received by the perceptionmodules 210), and then iteratively propagate its excitement out to nodeslinked to this particular node. Accordingly, as data (e.g., phase-spaceand/or primitive event symbol streams, low-dimensional vectors) arereceived from the semantic representation module 205, nodes of theneuro-semantic network may become activated. In one embodiment, nodesreaching an activation threshold, and possibly the associated codelets,are copied to the workspace 725. The copied nodes typically representconcepts of various complexity, for example, simple concept such as anobject or its characteristic, or complex concepts such as behaviors.

In general, the episodic memory 710 stores short-term data describingthe observed primitive events and/or behaviors. In other words, theepisodic memory 710 is a memory for storing recently perceived events(referred to as percepts). For example, a percept that has been recentlyacted upon by one or more codelets may be copied to the episodic memory710. Percepts in the episodic memory are typically specific and containinformation about “what,” “where,” and “when” for a particular observedevent and/or behavior. In one embodiment, the episodic memory 710 may beimplemented as a content-addressable memory. As is known, acontent-addressable memory (also referred to as associative memory) is atechnique for storing information that allows information to beretrieved based on content, not on a storage location (e.g., a memoryaddress) used to store content.

In contrast, the long-term memory 715 captures long-term data describing(or generalizing) events and/or behaviors observed in the scene. Thedata stored as memories in the long-term memory 715 are typically moregeneral (abstract) than those stored in the episodic memory 710. Forexample, in one embodiment, the long-term memory 715 is configured tomerge specific structures (i.e., percepts) into generalized abstractgroups. Thus, the long-term memory 715 may be used to build andaccumulate general events/patterns of behavior within a given scene. Inone embodiment, long term memory may be implemented as a sparsedistributed memory structure.

Additionally, events and/or patterns of behavior stored in the episodicmemory 710 that have survived for a substantial period of time may beused to develop or update a memory in the long-term memory 715. However,data stored in the long-term memory 715 may decay over time (e.g., thespecific details of an event may decay out of long-term-memory 715). Forexample, if several cars have been observed to park in the same placeover a period of time, then over time, a long term memory may developrepresenting a general pattern of a car being able to park in thatspecific location (or more generally, a car may park in a locationhaving the general characteristics associated with the actual observedlocation). At the same time, details regarding any particular parked carmay decay from the episodic memory 710 without ever reaching thelong-term memory 715, such as percept representing a one-time ordinaryevent (not abnormal as defined by the behavior recognition system).Techniques for reinforcing (or decaying) information stored in theepisodic memory 710 and the long-term memory 715 are described below.

In one embodiment, the workspace 725 uses information found in thelong-term memory 715 and/or the episodic memory 710 to analyzeevents/behaviors currently observed in the scene. When a percept isplaced into the workspace 725 from the perceptual associative memory705, similar data (i.e., percepts) may be retrieved from the episodicmemory 710 and/or the long term memory 715 and copied into theworkspace. In one embodiment, to determine whether a certain percept issimilar to a percept in the workspace 725, similarity scores are definedfor the percepts in the episodic memory 710 and the long-term memory715. Percepts having a similarity score above a certain threshold wouldbe considered similar. The similarity scores and/or thresholds may varyfor different percepts and/or between the episodic memory 710 and thelong-term memory 715. By using data from the episodic memory 710 and thelong-term memory 715, the perception module 210 uses both specificdetail and abstract information related to a current event/behavior tobetter understand that event/behavior.

In general, the scheduler 720 acts as a repository for codelets andselects what codelet to execute at any given time. For example, thescheduler 720 may identify a match between percepts placed in theworkspace 725 and the codelets. In one embodiment, codelets are providedto the scheduler 720 from the perceptual associative memory 705 and/orby an outside user. When an appropriate set of inputs required for agiven codelet (e.g., a set of precepts) is available that codelet may beplaced in the workspace 725 and executed. When multiple codelets areavailable for activation, the scheduler 720 may randomly select whichcodelet to execute. In one embodiment, some codelets may be assigned anurgency value defining an activation priority for a given codelet, e.g.,a codelet defining a certain abnormal behavior may have a higheractivation priority than a codelet defining normal behavior. At anygiven moment, numerous codelets may be in activated state within theworkspace 725.

As discussed above, codelets are typically executable code pieces thatprocess data and perform specific tasks. Frequently, a codelet maydescribe and/or look for relationships between different percepts. Insuch a case, a codelet may be configured to take a set of input preceptsand process them in a particular way. For example, a codelet may take aset of input percepts and evaluate them to determine whether aparticular event has occurred (e.g., a car parking). Moreover, a codeletmay be configured to create and/or destroy percepts, strengthen a bondbetween two or more percepts, and so on. Furthermore, codelets may movebetween various components of the perception module 210. For example,codelets may exist in the perceptual associative memory 705, thescheduler 720, and/or the workspace 725. Codelets may run independentlyand/or parallel to one another.

Further the perception module 210 may use a variety of differentcodelets to “learn” from observed events, including, e.g., perceptioncodelets (i.e., for looking for input features and creating semanticevents); structure codelets (i.e., for connecting nodes or smallerstructures in the workspace); behavior codelets (i.e., for recognizingsequences of events associated with a given behavior); predictioncodelets (i.e., for determining expected events based on pastexperiences); expectation codelets (i.e., for looking for expectedoutcomes and indicating when such expected outcomes are not achieved);timekeeper codelets (i.e., for creating events if not disabled after aperiod of time); and so on. Codelets for recognizing anomalies may alsobe employed by the perception module 210. Such codelets may evaluatepercepts to identify when a given percept does not statisticallycorrelate with previously accumulated statistical data. In such case, anabnormal (or just simply new) behavior may be identified.

FIGS. 8A-8C are a flowchart illustrating a method 800 for analyzing,learning, and recognizing behaviors, according to one embodiment of theinvention. As shown, the method starts at step 802. More specifically,FIGS. 8A-8C illustrate a cognitive cycle for a machine learning engineconfigured to analyze and learn from behaviors in a sequence of videoframes. Steps 804, 806, and 808 represent steps for receiving datadescribing events/behaviors observed in the scene. More specifically, atstep 804 trajectory information is received about an object tracked inthe scene. As described above, this information may provide a variety ofcharacteristics 864 of a tracked object at a particular trajectory pointof that object. For example, this information may include object's typeas identified at a particular trajectory point; data identifying theobject, such as an identification number; the object's velocity and/oracceleration; time associated with the trajectory point, such as a framenumber and/or a time value as defined within the behavior recognitionsystem 100; the trajectory point description, such as pixelsrepresenting the trajectory point, location within the scene, etc.;and/or other quantitative characteristics of the object.

At step 806, primitive events 866 identified in the video stream arereceived. In one embodiment, each primitive event is associated with atleast one object that participates in that primitive event. At step 808,formal language vectors 866 are received. As described above, the formallanguage vectors 866 may be represented as low-dimensional vectorsdescribing complete trajectories of the objects tracked in the scene. Inone embodiment, a formal language vector is associated with an objectand includes kinematics of the object exhibited along the object'strajectory in the scene together with primitive events that the objecthas participated in.

At step 812, higher level behavioral concepts 672 are perceived. Asdescribed above, incoming data may excite some nodes of a neuro-semanticnetwork located in a perceptual associative memory 705. If the incomingdata provides an adequate stimulus, then a set of nodes representing ahigher level concept may become activated. At step 814, a percept (i.e.,the set of excited nodes representing a higher level concept excited bythe input to the perceptual associative memory) may be copied to theworkspace 725 for further analysis. Note, some concepts may takemultiple video frames to be activated (e.g. a vehicle turns), whileother concepts may be activated essentially instantaneously, i.e.,requires only a single video frame (e.g., a vehicle appearing in thescene).

At step 816, memories relevant to the percept copied into the workspaceat step 814 may be used to search and retrieve various memories 876(such as memories from an episodic memory 710 and a long-term memory715). In one embodiment, a similarity measure may be defined todetermine whether a concept stored as a memory is relevant to a perceptcopied into the workspace. In one embodiment, all relevant data issought. In another embodiment, only the most relevant data (e.g., asdefined by the similarity measure) is sought. The relevant data may besought at each concept level (e.g., complexity level), only on a certainconcept level, or a search-codelet may alternate between differentconcept levels every time it runs. The retrieved concepts are alsocopied to the workspace 725. In this manner, the workspace acquires datauseful in interpreting and comparing currently observed behaviors/eventsto past ones.

At step 818, a codelet may be selected to execute based on theinformation then in the workspace 725. As described above, codelets maybe configured to analyze and process data placed into the workspace 725to recognize, interpret, and analyze behaviors observed by thebehavior-recognition system 100. When multiple codelets are availablefor activation, a codelet that is activated to run its particular taskmay be picked randomly (or semi-randomly, as discussed above). Theselected codelet may be configured to apply model based reasoning, logicbased reasoning, and reasoning by analogy to information copied to theworkspace 725 to recognize behaviors and/or other events. Further, inone embodiment, codelets may build new structures, such as combine twoor more percepts into a complex concept, and/or supply their ownstructures and name the newly created structures. The name for a higherlevel concept may be determined, e.g., by combining labels of thecombined structures.

As described above, one type of codelet may be configured to determinewhether an anomaly has occurred. For example, an “anomaly detector”codelet may analyze data in the workspace 725 to compare currentobservations in the scene with patterns stored in the long-term memory715. If such a codelet determines that differences are significant, ananomaly event/behavior may be identified. In one embodiment,trajectories in the scene and associated data are saved in a supportvector machine. Such data may be used the “anomaly detector” codelet todetermine whether a currently observed trajectory is “normal” for thatenvironment. In another embodiment, a codelet for sampling velocity andacceleration evaluates velocity and acceleration data and determinestheir distributions for each type of trajectory objects (e.g., velocityand acceleration distributions are likely to differ for a vehicle and ahuman). After statistically sufficient distribution samples arecollected, currently observed data may be compared against suchdistributions to determine whether the currently detected speeds and/orvelocities of the tracked objects are “normal.”

At step 820, a codelet (or corresponding percepts and memories) may beselected and placed into a focus of attention data structure. At step822, an indication of percepts stored in the focus of attention may bebroadcast to other components of the behavior-recognition system 100, soappropriate actions may be taken. In one embodiment, percepts in thefocus of attention 880 are also stored in the episodic memory 710.Further, two types of the actions may be taken based on the broadcastdata; namely, internal and external actions. At step 824, one or moreinternal actions are selected and performed. Internal action may includecreating/updating procedures, concepts, models and/or events, plansand/or expectations and so on.

For example, in one embodiment, when a new behavior is observed, a newconcept may be created. Assume that a percept representing a two-caraccident has previously been learned. Then, when the behaviorrecognition system 100 observes a three car accident, the system 100would recognize the three car accident as a new behavior. Consequently,a new concept may be created using the two car accident as a base. Todefine this new concept, a new higher level node may be created in aperceptual memory of the system 100. In one embodiment, such a nodewould be conceptually related to the node representing the two caraccident and one extra car node. A label may be assigned to the new nodeby combining labels of the nodes it is constructed from (e.g.,“car-car-car accident,” where “car” is a label of the car node and“car-car accident” is a label of the two car accident.

In another embodiment, similarity learning is implemented. For example,when a percept representing a currently observed behavior/event (e.g.,parking event), is similar to another percept (another parking event),previously learned, an internal action performed in response to thebroadcast of such a percept may be creating a new percept which is theaverage of the current percept and the previously learned percept (suchas a percept having average of the deceleration values and the like). Inthis manner the neuro-semantic network grows, providing larger pool ofsamples of observed behaviors for future behavioral analysis.

In yet another embodiment, a model of a behavior based on accumulateddata may be created based on the accumulated data. Within such a model,predictions of what should happen, where, and how, may be made. When anevent/behavior correlating with the model is broadcast, the model may beupdated. In other words, the behavior recognition system usesenvironmental feedback to learn behaviors. For example, if the behaviorrecognition system 100 observes two cars approaching each other at ahigh speed, the behavior recognition system 100 (e.g., an expectationcodelet) relying on an accident model may predict an accident willoccur. However, if subsequently, no accident occurs when the receivedinput satisfies requirements of the expectation codelet, the accidentmodel could be adjusted (e.g., do not predict a crash until two vehiclesapproaching at a high speed are within a certain distance of eachother). In one embodiment, when the expected event does not occur, theexpectation codelet attaches to the contradictory percept (a perceptrepresenting what really happened), excites nodes of the perceptualmemory, and reaches the focus of attention, so the model could beupdated using the current observations. The model would continue to beupdated when the appropriate input is available.

At step 826, one or more external actions may be performed. The externalactions may generally include any action that involves communicatingwith something (or someone) outside of the machine learning engine. Forexample, external actions may include issuing alarms (e.g., soundsindicating abnormal event/behavior, fire alarm, etc.), messages (e.g.,printing a message on a screen, sending e-mail, sending a text messageover the phone, calling police, etc.), adjusting operation of a videoacquisition device (e.g., adjusting contrast in a video camera, view ofthe video camera, etc.), combination thereof, and so on. Different typesof observations (e.g., abnormal event, specific event, etc.) may causedifferent actions to be performed. For example, a particular observationmay be associated with a specific external action or set of externalactions (e.g., plan of actions); an external action to be performed maybe determined using previous experience; default action may be selected;etc. In one embodiment, the external actions may also include providingfeedbacks to the semantic representation module 205 and/or computervision engine 135. Furthermore, the external actions may be pre-defined,learned, or both. Moreover, in one embodiment, the external action maybe modified via an outside input.

Steps 828 through 844 illustrate an example of decay/reinforcementprocedures that may be used by the behavior recognition system 100. Ingeneral, information (such as percepts and/or memories in the episodicor long-term memories) useful to interpret observed behaviors isreinforced by increasing its base activation, while a//informationstructures decay by lowering it. If the base activation of a percept orother structure decays below a threshold, it is eliminated. At step 828,a behavioral structure/procedure is selected to determine whether itshould be reinforced. Typically, percepts that reach the focus ofattention are reinforced. In other words, percepts observed frequentlysurvive while others do not. At step 830, it is determined whether theselected behavior/procedure has reached the focus of attention. If theselected structure/procedure has reached the focus of attention, themethod 800 proceeds with step 832, where it is determined whether theselected structure/procedure is an expectation procedure.

Expectation procedures typically predict how a certain behavior/eventwould progress. Such a prediction may be correct, and thus, be useful inanalyzing future behaviors/events, or incorrect, and thus, may sabotageproper analysis of future behaviors/events. Accordingly, in oneembodiment, when the expectation procedure is incorrect in itspredictions, i.e., with proper inputs the expected result did not occur(determined at step 836), the base activation value of the procedure islowered at step 838, i.e., the procedure decays. However, if theexpectation procedure is correct and the expected result did occur, thenthe procedure is reinforced via increasing the base activation of theprocedure at step 834. Note, however, in one embodiment, even though amemory is reinforced at step 824, it may also be lowered as part of step838 by a different (typically lower) amount. That is, all memorystructures may decay, but only some are reinforced. Thus, structures(e.g., behaviors and procedures) adequately reinforced will remain,while others may ultimately decay away. Behaviors/procedures that havereached the focus of attention and are not expectation procedures aresimilarly reinforced at step 834.

A base activation generally represents how permanent and/or accurate isa particular structure/procedure. By how much a base activation value isincreased/lowered in each particular case may be determined by a varietyof factors, such as where the structure/procedure is found (e.g.episodic memory vs. long term memory), type of structure/procedure(e.g., normal vs. abnormal, simple v. complex, etc.), and so on. In oneembodiment, there are two kinds of decay procedures which areimplemented in the behavior recognition system 100. One kind is a lineardecay that is applied, for example, to the content of the workspace 725.Another kind is non-linear decay that is applied, for example, to thestructures/procedures stored in the memories. In this manner,structures/procedures found in the workspace decay quickly unless theyreach the focus of attention. In contrast, once structures/proceduresreach one of the memories and receive sufficient reinforcement, theydecay at a slower rate.

Furthermore, different components of the behavior recognition system mayhave different decay and/or reinforcement rates. For example, in oneembodiment, different memories have different decay rates, e.g., theepisodic memory's decay rate is higher (structures/procedures decayfaster) than the long term memory's decay rate (structures/proceduresdecay slower). In another embodiment, structures/procedures placed inone of the behavior-recognition system's components never decay. Forexample, codelets found in the scheduler 720 may never decay.

Moreover, different structures/procedures may have different rate ofreinforcement and/or decay. For example, abnormal events/behaviors, suchas violent interactions, usually do not happen frequently. Consequently,percepts or codelets associated with the abnormal behaviors do not reachthe focus of attention as often. However, it may be beneficial to keepdata describing the abnormal behaviors/events available. Accordingly, inone embodiment, the decay rate for the abnormal events/behaviors is verylow. In another embodiment, special codelets are employed to reinforcethe abnormal behaviors/structures even when they do not reach the focusof attention. In yet another embodiment, data associated with theabnormal behaviors/events simply does not decay.

As structures/procedures decay, they may eventually become eliminated.In one embodiment, at step 840, the activation base value of theselected structure/procedure is compared to a pre-defined removalthreshold to determine whether the structure/procedure needs to beeliminated. If the activation base value is equal or below the removalthreshold then, at step 842, the structure/procedure is eliminated. Notethat similar to the decay/reinforcement rates, a pre-defined removalthreshold may vary for different structures/procedures and/or thebehavior recognition system's components. At step 844 it is determinedwhether the decay/reinforcement procedure has been completed. If yes,then the method 800 returns to step 802 and initiates another cycle ofthe cognitive process.

However, if the decay/reinforcement procedure has not been completed(e.g., not every structure/procedure has been selected forreinforcement/decay) the method 800 returns to step 828, where a newstructure/procedure is selected.

In one embodiment, the reinforcement/decay procedure is implementedusing various codelets. In another embodiment, only the reinforcement ofthe structures/procedures is implemented using codelets, while the decayis included into a main loop of the behavior recognition system 100(e.g., the method 300 illustrated in FIG. 3). Note however, that it isnot necessary to perform all of the above-described steps of the method800 in the order named. Furthermore, not all of the described steps arenecessary for the described method to operate. Which steps should beused, in what order the steps should be performed, and whether somesteps should be repeated more often than other steps is determined,based on, for example, needs of a particular user, specific qualities ofan observed environment, and so on.

Advantageously, as described herein, embodiments of the invention enablerecognizing and learning newly perceived objects and behaviors and theirrelationship to already known objects and behaviors within an observedenvironment. Moreover, embodiments of the invention enable usingenvironmental feedback for accurately evaluating, reinforcing, andmodifying the patterns of behaviors learned about a given object.Furthermore, embodiments of the invention enable identifying which ofthe observed behaviors are normal or abnormal. Also, embodiments of theinvention enable reinforcing repeatedly occurring behaviors whiledecaying memories representing behaviors that only occur occasionally.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed:
 1. A computer-implemented method for processing datarelated to a scene depicted in a sequence of video frames, the methodcomprising: receiving input data describing an object detected in thescene, wherein the input data comprises at least a classification forthe object; identifying one or more primitive events, related to abehavior engaged in by the object detected in the scene and wherein eachprimitive event has an assigned primitive event symbol; generating, forthe object, a primitive event symbol stream comprising the primitiveevent symbols corresponding to the primitive events identified for theobject; forming a first vector representing the object; and comparingthe first vector to a stored vector to define a similarity measure. 2.The computer-implemented method of claim 1, further comprising, applyinga singular value decomposition (SVD) to the first vector to generate asecond vector from the first vector, wherein the second vector reducesthe dimensionality of the first vector.
 3. The computer-implementedmethod of claim 1, wherein the classification for the object specifiesthat the object depicts one of a vehicle object, a person object, or anunknown object.
 4. The computer-implemented method of claim 3, whereinthe object is classified as a person, and wherein the input data furtherincludes a posture of the person as depicted in the scene.
 5. Thecomputer-implemented method of claim 3, wherein the object is classifiedas a vehicle, and wherein the input data further comprises behaviorengaged in by the vehicle that includes one or more of appearing,moving, turning and stopping.
 6. The computer-implemented method ofclaim 3, wherein the object is classified as a person, and wherein theinput data further comprises behavior engaged in by the person thatincludes one or more of appearing, moving, turning and stopping.
 7. Thecomputer-implemented method of claim 3, wherein the object is classifiedas an unknown object, and wherein the input data further comprisesbehavior engaged in by the unknown object that includes one or more ofappearing, moving, turning and stopping.
 8. The computer-implementedmethod of claim 1, wherein comparing includes comparing the first vectorto the stored vector with a machine learning engine.
 9. A non-transitorycomputer-readable medium containing a program, which, when executed on aprocessor is configured to perform an operation for processing datarelated to a scene depicted in a sequence of video frames, comprising:receiving input data describing an object detected in the scene, whereinthe input data comprises at least a classification for the object;identifying one or more primitive events, related to a behavior engagedin by the object detected in the scene and wherein each primitive eventhas an assigned primitive event symbol; generating, for the object, aprimitive event symbol stream which includes the primitive event symbolscorresponding to the primitive events identified for the object; forminga first vector representing the object; and comparing the first vectorto a stored vector to define a similarity measure.
 10. Thenon-transitory computer-readable medium of claim 9, further comprising,applying a singular value decomposition (SVD) to the first vector togenerate a second vector from the first vector, wherein the secondvector reduces the dimensionality of the first vector.
 11. Thenon-transitory computer-readable medium of claim 9, wherein theclassification for the object specifies that the object depicted in thescene depicts one of a vehicle object, a person object, or an unknownobject.
 12. The non-transitory computer-readable medium of claim 11,wherein the object is classified as a person, and wherein the input datafurther includes a posture of the person as depicted in the scene. 13.The computer-implemented method of claim 9, wherein comparing includescomparing the first vector representation with a machine learningengine.
 14. A system, comprising: a video input source; a processor; anda memory storing computer instructions, which, when executed on theprocessor configure the processor to: input data describing a scenedepicted in a sequence of video frames, the input data comprises atleast a classification for an object in a scene; identify one or moreprimitive events, wherein each primitive event provides a semantic valuedescribing a behavior engaged in by the object depicted in the scene andwherein each primitive event has an assigned primitive event symbol;generate, for the object, a primitive event symbol stream whichcomprises the primitive event symbols corresponding to the primitiveevents identified for the object; form a first vector representing theobject; and compare the first vector to a stored vector to define asimilarity measure.
 15. The system of claim 14, wherein the processor isfurther configured to apply a singular value decomposition (SVD) to thefirst vector to generate a second vector from the first vector, whereinthe second vector reduces the dimensionality of the first vectorrepresentation.
 16. The system of claim 14, wherein the classificationfor the object specifies that the object depicted in the scene depictsone of a vehicle object, a person object, or an unknown object.
 17. Thesystem of claim 16, wherein the object is classified as a person, andwherein the input data further includes a posture of the person asdepicted in the scene.
 18. The system of claim 16, wherein the object isclassified as a vehicle, and wherein the behavior engaged in by thevehicle includes one or more of appearing, moving, turning and stopping.19. The system of claim 16, wherein the object is classified as aperson, and wherein the behavior engaged in by the person includes oneor more of appearing, moving, turning and stopping.
 20. The system ofclaim 14, wherein a machine learning engine is employed to compare thefirst vector to the stored vector.